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ABSTRACT 

Nonlinear redshift-space distortions, the Finger-of-God (FoG) effect, can complicate the in- 
terpretation of the galaxy power spectrum. Here, we demonstrate the method proposed by 
Hikage et al. (2012) to use complimentary observations to directly constrain this effect on the 
data. We use catalogs of Luminous Red Galaxies (LRGs) and photometric galaxies from the 
Sloan Digital Sky Survey (SDSS) Data Release 7 (DR7) to measure the redshift-space power 
spectrum of LRGs, the cross-correlation of LRGs with the shapes of background photometric 
galaxies (galaxy-galaxy weak lensing), and the projected cross-correlation of LRGs with pho- 
tometric galaxies having similar photometric redshifts to the LRG spectroscopic redshift. All 
of these measurements use a reconstructed halo field. While we use the position of each LRG 
for single LRG systems, we compare the measurements using different halo-center proxies 
for multiple-LRG systems (4.5 per cent of all the halos): the brightest LRG position (BLRG), 
the faintest LRG position (FLRG) and their geometrical mean position (Mean), respectively, 
in each system. We find significant differences in the measured correlations of different cen- 
ters, showing consistent off-centering effects in the three observables. By comparing the mea- 
surements with the halo model, we find that ~ 40 (70) per cent of BLRGs (FLRGs) are 
off-centered satellite galaxies in the multiple-LRG systems. The satellite LRGs have typical 
off-centering radius of ^400 kpc/h, and velocity dispersion of about 500 km/s in host halos 
with a mean mass of 1.6 x 10 14 M Q / h. We show that, if LRGs in the single LRG systems have 
similar offsets, the residual FoG contamination in the LRG power spectrum can be significant 
at k ^ 0.1 h/Mpc, which may cause a bias in cosmological parameters determined by the 
shape of the power spectrum, such as the neutrino mass. Our results demonstrate that over- 
lapping spectroscopic and imaging galaxy surveys can be used to observationally calibrate 
the FoG contamination and more robustly use the galaxy power spectrum for cosmological 
measurements. 

Key words: cosmology: theory and observations - galaxy clustering - gravitational lensing: 
weak - large-scale structure of the Universe - cosmological parameters 



1 INTRODUCTION 

Large-scale structure surveys are one of the primary tools for studying the nature of dark matter, dark energy and the initial conditions in 
the early universe (Davis & Huchra 1982; de Lapparent et al. 1986; Kirshner et al. 1987; York et al. 2000b; Peacock et al. 2001). Over 
the coming decade, astronomers are embarking on even larger surveys including the Baryon Oscillation Spectroscopic Survey (BOSS 1 , 



1 http://cosmology.lbl.gov/BOSS/ 
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Dawson et al. 2012), WiggleZ 2 (Blake et al. 2011), Vipers 3 , FMOS 4 , HETDEX 5 , BigBOSS 6 (Schlegel et al. 2009), Subaru Prime Focus 
Spectrograph (PFS 7 ; Ellis et al. 2012), Euclid 8 , and WFIRST 9 . The upcoming generation of surveys is motivated by our desire to understand 
cosmic acceleration and to measure the composition of the universe by simultaneously measuring geometry and dynamics (Wang et al. 1999; 
Eisenstein et al. 1999; Tegmark et al. 2004; Cole et al. 2005). Measurements of the baryon acoustic oscillation (BAO) scale provide us with 
a robust geometrical probe of the angular diameter distance and the Hubble expansion rate (Eisenstein et al. 2005; Percival et al. 2007; 
Anderson et al. 2012). Observations of redshift-space distortion measure the growth rate of structure formation (Zhang et al. 2007; Guzzo 
et al. 2008; Wang 2008; Guzik et al. 2010; White et al. 2009; Percival & White 2009; Song & Percival 2009; Song & Kayo 2010; Yamamoto 
et al. 2010; Tang et al. 2011; Reid et al. 2012). Combining measurements of the geometry of the universe and the growth of structure 
formation provides a key clue to understanding the nature of dark energy as well as constraining properties of gravity on cosmological scales 
(Albrecht et al. 2006; Peacock et al. 2006). 

If the distribution of galaxies in redshift space simply traced the underlying distribution of matter in real space, then cosmologists could 
easily interpret the large-scale structure observations. The universe, however, is not so simple: galaxies do not necessarily sit in the centers 
of dark matter halos and correspondingly, they are moving in these halos. This Finger-of-God (FoG) effect (Jackson 1972; Peacock & Dodds 
1994; Scoccimarro 2004) is a significant source of systematic uncertainty in the cosmological interpretation of the redshift-space galaxy 
power spectrum 

Reid et al. (2009) advocated using halos rather than Luminous Red Galaxies (LRGs; Eisenstein et al. 2001b) to mitigate FoG effects. 
In an analysis of LRGs from the Sloan Digital Sky Survey 10 (SDSS) Data Release 7 (DR7) catalog, Reid et al. (2010) implemented this 
scheme by removing satellite LRGs from within halos hosted by LRGs, with the aid of mock catalogs and the halo model prescription. Once 
such a halo catalog is constructed, the clustering properties of halos are easier to model, because halos have only bulk motions in large-scale 
structure, and therefore have a reduced FoG effect. However, even if one can perfectly choose one galaxy in each halo (e.g., the brightest LRG 
or the brightest cluster galaxy), there still remains uncertainty; the position of the chosen galaxy is not necessarily at the halo center (Skibba 
et al. 2011), and the off-centered galaxies have an internal motion relative to the true halo center, which causes residual FoG contamination 
(Seljak 2001; White 2001; Hikage et al. 2012). 

In our previous paper (Hikage et al. 2012), we developed a method to correct the residual FoG effect by combining the redshift-space 
galaxy power spectrum with the cross-correlation of spectroscopic galaxies with images of background galaxies - the so-called galaxy-galaxy 
weak lensing (e.g., Mandelbaum et al. 2006b; Sheldon et al. 2009; Leauthaud et al. 2010; Okabe et al. 2010). The galaxy-galaxy lensing 
measures the average mass distribution around spectroscopic galaxies; i.e., at small scales, it reveals the mass profile of the halos hosting the 
lens galaxies, while on large scales it is sensitive to the mass distribution surrounding the host halos. However, if we include off-centered 
galaxies and use their positions as a proxy for the center of each halo, the lensing signals at angular scales smaller than the typical offset scale 
are diluted. In other words, comparing the galaxy-galaxy lensing signals of different halo-center proxies can be used to infer the amount of 
the off-centered galaxy contamination (Johnston et al. 2007; Leauthaud et al. 2010; Okabe et al. 2010; George et al. 2012, also see Oguri & 
Takada 201 1 for a useful formulation of the off-centering effect on galaxy-galaxy weak lensing). Hence, a weak-lensing based calibration 
of the FoG effect in redshift-space power spectrum measurements can be feasible if spectroscopic and imaging surveys observe the same 
region of the sky. Fortunately, this is the case for many upcoming surveys: the BOSS and the Subaru Hyper Suprime-Cam (HSC) Survey 11 
(Miyazaki et al. 2006), the Subaru PFS and HSC surveys (Subaru Measurements of Images and Redshifts: the SuMIRe project), Euclid and 
WFIRST or a combination of LSST (LSST Science Collaborations et al. 2009) with spectroscopic surveys. 

The goal of this paper is to implement this method and to determine how LRGs are distributed in dark matter halos. For this paper, we use 
the SDSS DR7 catalog to constrain the fraction of off-centered LRGs and the amount of the residual FoG contamination in the LRG power 
spectrum measurement. We focus on systems that contain multiple LRGs in the same halo, defined based on the friend-of-friends algorithm 
(Reid & Spergel 2009; Reid et al. 2010). Then, we compare the redshift-space power spectra and the cross-correlations of LRGs with shapes 
of background photometric galaxies, measured using different halo-center proxies. Furthermore, we also use the projected cross-correlations 
of LRGs with photometric galaxies that have similar photometric redshifts to the LRG redshift, and then compare the measurements using 
different halo centers. We employ the following three proxies for the halo center: the brightest LRG position, the faintest LRG position, and 
the geometrical mean position of LRGs in each multiple-LRG system. We show that comparing the measurements to the halo model allows 
us to constrain the population of off-centered LRGs, the amount of off-centering effects, and the residual FoG contamination. 

The structure of this paper is as follows. In Section 2, we develop a halo model to compute the redshift-space power spectrum, LRG- 
galaxy lensing and LRG-photometric galaxy cross-correlation including off-centering effects. In Section 3, we show the power spectrum, 
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galaxy-galaxy lensing and the photometric galaxy cross-correlation measured from the SDSS DR7 catalog of LRGs and photometric galaxies, 
using the different halo-center proxies. Then we constrain the model parameters by comparing the halo model predictions to the measure- 
ments. Based on these findings, in Section 4, we discuss the residual FoG contamination in the LRG power spectrum, and its impact on 
cosmological parameters. Section 5 is devoted to discussion and conclusions. Unless explicitly stated otherwise, throughout this paper we 
assume a WMAP-normalized ACDM model as our fiducial cosmological model (Komatsu et al. 2009): fi D /i 2 = 0.0226, 0. c d m h 2 = 0.1109, 
Oa = 0.734 r = 0.088, n s = 0.963, and as = 0.817, where fib, ficdm and fiA are the energy density parameters of baryon, CDM and 
dark energy (the cosmological constant with wo = — 1 here), r is the optical depth to the last scattering surface, and n s is the tilt of the 
primordial curvature power spectrum. 



2 THE EFFECTS OF OFF-CENTERED LRGS ON THE GALAXY CLUSTERING AND WEAK LENSING CORRELATIONS 

To constrain the radial distribution of LRGs within their host halos, we use three observables that can be measured from the SDSS DR7 
data (Abazajian et al. 2009b): the angle-averaged, redshift-space power spectrum of LRGs, the weak lensing correlation of LRGs with 
background galaxy images, and the projected cross-correlation of LRGs with photometric redshift galaxies that have photo-2's similar to the 
LRG redshift. When computing these correlation functions, we use the same LRG catalog, but compare the correlation functions measured 
using different halo-center proxies (see below). Before describing the measurements, we first describe our halo model for the correlation 
functions, including the effect of off-centered LRGs (also see Oguri & Takada 201 1; Hikage et al. 2012). 

2.1 Connecting LRGs to halos and the radial offset distribution of LRGs 

Weak lensing studies (e.g., Mandelbaum et al. 2006b; Johnston et al. 2007) and clustering analyses (Ross et al. 2007, 2008; Wake et al. 2008; 
Zheng et al. 2009; Reid & Spergel 2009; White et al. 201 1) have shown that most LRGs reside in massive halos of a few x 10 13 M o . While 
the typical massive halo contains only one LRG, about 5 per cent of LRGs are found to be satellite galaxies in a halo; there is a population of 
halos that contain multiple LRGs (Reid & Spergel 2009). Including these satellite galaxies in the correlation function measurements alters the 
clustering signals at small scales. This contribution complicates the estimation of cosmological parameters, because the small-scale signals 
involve complicated physical processes of galaxy formation/evolution that are still very challenging to accurately model. Hence, in this paper, 
we employ a method similar to that developed in Reid & Spergel (2009) to connect the distribution of LRGs to that of host halos by removing 
satellite LRGs from the catalog (see also Reid et al. 2010). In other words, we assume that we can identify one LRG (or one halo-center 
proxy) in each LRG system. 

Since we are interested in the LRG-related correlation functions, which are statistical quantities, we need to model the average radial 
distribution of LRGs within host halos as a function of halo mass M: p s(r), which is normalized as 47r J" v " r 2 drp fi(r) — 1. If LRGs 
are at the halo center, p s(r) = <5i)(r)/47rr 2 . For off-centered LRGs, the Fourier transform of the radial profile is given as 

/, „ ,n f Tv " 2, / .sin Ax 

PoB (k; M) = 4ir J rdrp off (r) — , (1) 

where we will often omit the halo mass dependence of p Q ff for notational simplicity. For central LRGs, p a s(k) = 1. In this paper, we assume 
for simplicity that offset LRGs follow a Gaussian distribution, p g(r) oc exp[— r 2 /2R 2 oS ], and treat R off as a free parameter to constrain 
from the measured correlation functions. However, the exact shape of the p s(r) is less important than its characteristic width. 

2.2 The FoG effect on the redshift-space power spectrum of LRGs 

As we discussed in detail in Hikage et al. (2012), the major effect of off-centered LRGs on the angle-averaged, redshift-space power spectrum 
of LRGs is the nonlinear redshift-space distortion effect, the so-called Finger-of-God (FoG) effect. If an LRG is offset from the true halo 
center, the LRG should be in motion with respect to it. This internal motion within halos causes the FoG effect on the LRG power spectrum, 
and can occur even if selecting one LRG (or one center more generally) per halo, provided that the selected LRG or center is not the center 
of mass of the halo. By employing the halo model approach, we can model the redshift-space LRG power spectrum at a redshift z as 

2 

-P S N m(fc,M), (2) 

where /j, is the cosine of the angle between the line-of-sight direction and the wavevector k, i.e. /i = ku/k; dn/dM is the halo mass 
function of halos with mass M and at redshift z; b(M) is the linear halo bias function; Nhod(M) is the halo occupation distribution 
(HOD) specifying the probability that halos of mass M host LRGs (A^hod ^ 1 in our setting); wlrg is the mean number density of LRGs 
defined as «lrg = /dA / /(dn/dM)A'HOD(M); P^m is me nonlinear redshift-space power spectrum of dark matter. The offset function 
Ps,o«{k, n) is defined by convolving the real-space offset profile p ff (Eq. 1) with the velocity function along the line-of-sight direction via 
the virial theorem. See Hikage et al. (2012) for details. Although we employ the linear halo bias in the above equation, which is not a good 
approximation for a high-precision measurement in the quasi-nonlinear scales (Baldauf et al. 2010), this is not crucial for the following 



,LRG 



(fc,£i) 



— — [dM ^-b(M)Nnon(M)p a>off (k,fi;M) 
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results, because we will focus on fractional differences between the redshift-space power spectra of different halo center proxies, where the 
bias factor or the dark matter power spectrum cancel out in the fractional difference. 

To make the model more general, we introduce an additional parameter q ccn (0 ^ q ccn ^ 1) to represent the fraction of central LRGs 
among all the LRGs in the sample: the fraction of central LRGs is q C en, while 1 — q can is the fraction of satellite LRGs. In this case, we can 
model the resulting redshift-space power spectrum by replacing p s , a in Eq. (2) as 

Ps.off (k, fj,) ► (Jccn + (1 — qccn)Ps,os{k, fl). (3) 

Here we assumed that, for the first term, the central galaxies have no FoG suppression, i.e. are at rest at the center of the host halo, so that 
their p Sj0 fi = 1. 

In this paper, we consider the angle(/i)-averaged power spectrum defined as 

Ps,LR G (k)= J ^P s ,LR G (fc,At). (4) 

For notational simplicity, we will hereafter omit the notation " ~ " of P s ,LHG(k). 



2.3 The effect of off-centred LRGs on LRG-galaxy weak lensing 

By cross-correlating the positions of LRGs with shapes of background galaxies, we can measure the average dark matter distribution around 
LRGs - the so-called LRG-shear cross-correlation or LRG-galaxy weak lensing (Mandelbaum et al. 2006b, 2012b). If we include off-centered 
LRGs in the analysis, they dilute the measured dark matter profile at scales smaller than the off-centering radius (Johnston et al. 2007; Oguri 
& Takada 201 1; Hikage et al. 2012). Under the assumption that the dark matter halos are well-described by an NFW profile (Navarro et al. 
1996) or some other parametrized profile, we can use the measured surface density profile to infer the distribution of off-centered halos. 
For the observable of the LRG-galaxy weak lensing, we use the projected mass profile measured as a function of the projected radius: 

AE(i?) = / ^C Sg (k)J 2 (kR), (5) 

where J2 (x) is the 2nd-order Bessel function; R is the projected separation between LRG and background galaxy in units of the comoving 
scale; Cs g (k) is the angular power spectrum of the projected mass and LRG cross-correlation, and we give the power spectrum as a function 
of the wavenumber k for convenience in the rest of our discussion. Here the projected radius needs to be estimated from the observed angular 
separation for each LRG-galaxy pair on the sky: R — x@, where \ is the comoving radial distance to each LRG. This conversion requires 
a redshift for each lens LRG, which is available from the SDSS DR7 catalog, and the assumption of a background cosmological model, for 
which we assume the WMAP cosmology (Komatsu et al. 2011). The .R-average preserves the physical scales of host halos, such as the virial 
radius. On the other hand, if the cross-correlation is done in terms of the angular separation, then different scales are mixed for the halos at 
different redshifts, causing a smearing of the halo mass profile. 

Employing the flat-sky approximation and the halo model, we can express the power spectrum Cs g (k) as a sum of the 1- and 2-halo 
terms: 

Cx 3 (k) = C^(fc) + C| h 9 (fc). (6) 

The 1-halo term arises from the mass distribution within one halo that hosts LRGs and is the dominant contribution to the signal at small 
radii, while the 2-halo term arises from the mass distribution surrounding the host halos and is the dominant contribution at large radii. Using 
Limber's approximation (Limber 1954), the 1- and 2-halo terms are given as 

Ch h g (k) = /dx/ L RG(x) [ &M ^Nkoo(M; z) [Mu NFW (k; M, z)p oS (k; M) + m sh , LRG ] , 



C|g(fc) EE /dx/LRG(x) 



dM-^b(M;z)Nko%(M;z) 



PmO-PmO; Z ) , ( 7 ) 



where UNFw(k; M, z) is the Fourier transform of an NFW profile with mass M at redshift 2; m s h,LRG is a parameter representing the 
typical mass of a sub-halo hosting a satellite LRG; p m o is the mean mass density today; and Pm(k) is the linear mass power spectrum (see 
Hikage et al. 2012, for details). Our use of the mean mass density today (p m o) rather than at the LRG redshift is due to our use of comoving 
units throughout this paper. The definition of the power spectra differs from the usual definition by a factor of \~ 2 ( e -g- see Eqs. 24 and 
25 in Hikage et al. 2012, for comparison), arising from the conversion R O 9 discussed above. In the following analysis, we will focus on 
the galaxy-galaxy lensing correlations at scales up to a few Mpc, where the 1-halo term is dominant. However, as a conservative approach, 
we also include the 2-halo term treating the bias parameter as a free parameter, and then derive constraints on the off-centering parameters, 
marginalized over other parameter uncertainties. The function /lrg(x) m Eq. (7) is the radial selection function of LRGs defined as 

/lrg(x) = _ 2D »Slrg(z), 
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nllc = J d x S LRG (x) J dM^Nk%6(M), (8) 

where Slr,g(z) is the redshift selection function of LRGs, n™ G is the mean surface number density of LRGs, and the selection function 
satisfies the normalization condition J dx/LRG(x) J dM(dn/dM)Ni,nG(M) — 1. For simplicity, we assumed that the sub-halo lensing 
contribution can be characterized by a single number, a point mass, with a profile AE(i?) = m s h,LRG/(7i~^ 2 ). See Section 3.2 in Takada & 
Jain (2003) for the conversion relation between the three-dimensional and projected mass density profiles in computing the projected power 
spectrum in a halo model formulation. 

To gain insight into Eq. (7), let us consider a case that the LRGs are in a thin redshift slice centered at zlrg, and reside in halos with a 
narrow mass range centered at M. In this case, the power spectra can be simplified as 



Ch h 3 (k) ^ [A4m FW (k;M,z- LRG ) p oH (k;M) + m shiLRG ] 

~<2h i 



Cf£(fc) ^ b(M)p m0 P^(k;z). (9) 



These equations explicitly show that the 1-halo term arises form the NFW halo profile, and the 2-halo term probes the power spectrum with 
bias factor. When inverse Fourier-transforming the equations above, we can obtain the 1- and 2-halo term predictions for the projected mass 
profile AE(i?) to compare with the measurement. The above equations also show that the dimension of the power spectrum Cs g (k) is in 
units of [Af©]; the LRG-shear correlation is proportional to k 2 C^ g (k) (see Eq. 5), which has dimensions of surface mass density in units of 
[Mq /Mpc 2 ]. Given statistical uncertainties of the measurements for a small sample size of the multiple-LRG systems, we will use the above 
equations to constrain the averaged halo parameters and off-center parameters for the multiple-LRG systems. 

When we consider the case that only some fraction of LRGs have offsets, and the remaining LRGs are central galaxies in their host 
halos, we replace the mass profile in the above equation with 

Mu NFW (k; M, 2lrg) Pos(k; M) — > [q ccn + (1 - q cen )p a{k; M)] Mii NFW (k; M, zlrg) , (10) 

where g con is the fraction of central LRGs in the sample, and we assumed that all the LRGs reside in halos of the same (or similar) mass scale 
M . In this case, the sub-halo lensing contribution can be considered as the mean mass of sub-halos hosting the central and satellite LRGs. For 
the central LRGs, this enhancement relative to the NFW lensing of host halos might be considered to model additional lensing contribution 
due to baryonic contraction at the halo center. However, the sub-halo and baryonic contraction contributions are generally degenerate, unless 
we know which LRGs in the catalog are central galaxies. Hence the sub-halo contribution in the model (Eq. 9) can be considered as the 
average of the two effects. 



2.4 The projected cross-power spectrum of off-centered LRGs with photometric galaxies 

The third observable we consider is the projected cross-correlation function of positions of LRGs with positions of photometric-redshift 
galaxies, measured as a function of the projected separation as in the LRG-galaxy weak lensing (see Eq. 5). Even in the presence of pho- 
tometric redshift errors (photo-z errors), the cross-correlation method is very powerful in a sense that it can statistically discriminate, from 
photo-z outliers, photometric galaxies that actually cluster with LRGs at the same redshift 12 (Newman 2008). While including photo- z 
outliers in the analysis dilutes the cross-correlation amplitude, it does not alter the shape of the correlation function. 

Similar to the formalism for the LRG-galaxy lensing signal, the projected cross-correlation function can be expressed in terms of the 
power spectrum as 



■(B) = / ^cZ OSB (k)MkR), 



where the notation "cross" is introduced to explicitly mean the cross-correlation between different populations of galaxies, spectroscopic 
LRGs and photometric galaxies, not the auto-correlation function of the same galaxy population. Based on the halo model, the galaxy power 
spectrum is given as C^° BB (k) = C" g OBB ' lh (k) + C c g r ° BB - 2h (k) with the 1- and 2-halo terms being defined as 



C c ; g ° BB - lh (k) = I d x / L R G (x)/phg(x) / dM ^jv£gg(M)<^ D (M)fi NF w(fc; M, z)p oB (k; M), 



C gg OBB ' 2h {k) ee / dx/ L RG(x)/ P hg(x) 



dM ^N^%(M)b(M) 



dM ' ^p N *o»(M')b(M') 



(A; z) , (12) 



where N^on (M) i s me na l° occupation distribution for photometric galaxies, and we have assumed that the radial profile of photometric 
galaxies within the host halos follows an NFW profile for simplicity. We again note that, in the following results, we will focus on the 
cross-correlation at scales up to a few Mpc, so our use of the linear power spectrum in the 2-halo term does not cause any serious systematic 



1 At small scales where lensing magnification is important, there might seem to be some additional clustering with photometric galaxies; this should also be 
modeled if those scales are used for the measurements (unlike in this work). 
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error. We will take into account the uncertainty in the 2-halo term by treating the bias parameter as a free parameter, as in the LRG-galaxy 
lensing case. Taking into account the photo-z errors, the redshift selection function / p h g is given as 

1 dz 

/phg(x) = —20 P{z\zphz)-r-, (13) 

n phg,all d X 

where p(z|z p h z )Az is the probability that photometric galaxies with photo-z of z p h z have a true redshift in the range [z— Az/2, 2+ A/2]. The 
probability satisfies the following normalization condition: Jdzp(z|z p h z ) = 1. However, generally p(z|z p hz) Az ^ 1 around the LRG red- 
shift, so the factor p(z I Zphz) Az represents the dilution effect that is caused by including photo-z outliers in the correlation analysis. The quan- 
tity Wp° g all is the mean surface number density of all the photo-z galaxies, defined as n^ g all = J dxf>(z|z P hz)(dz/dx) / dM(dn/dM)N^Q D (M). 
The selection function / p h g (x) satisfies the nor malization condition: Jdx/ P h g (x) JdM(dn/dM)N^ D (M) = 1. 
Assuming a thin redshift slice and narrow halo mass bin of LRGs, we can simplify the galaxy power spectra as 



lfi/.x 1 "phg 



Cgg (k) 



(zlrg) 



A ^ n 

X dM 



r? 2D 

AM Phg, all 



u NFW (k; M, z LRG ) p ff(fc; M), 



C gg( k ) - "a OLRGOphg-P™ ^lrg) , (14) 

n phg,all 

where Ax is the radial distance width of the LRG distribution, and ^^^(zlrg) is the average surface number density of photometric galaxies 
that indeed lie in the LRG redshift bin and reside in halos hosting LRGs, defined as ^^(zlrg) = p(^LRG|^ P hz) dz/dx| ZLRG Axdn/dM]^ AM. 
The dimension of the power spectra is [Mpc] 2 . The prefactor ^ph g ( z LRG)/Wp° gall gives the fraction of photometric galaxies with true red- 
shifts that are sufficiently close to the LRG redshift. Hence, this factor accounts for the dilution effect due to photo-z outliers that we include 
in our sample. Also note that the 1-halo term of the power spectrum behaves like the Poisson shot noise l/[Ax(dn/dM)AA/] in the lim- 
its of p ff,UNFw — > 1, where Ax(dn/dM)AM is the mean surface number density of halos hosting LRGs. In the following analysis, 
we introduce a nuisance parameter to model the amplitude uncertainty, and will derive constraints on the off-centering profile of LRGs, 
marginalizing over the nuisance parameter. When further including the mixture of central and satellite LRGs in the sample, we will use the 
replacement given in Eq. (10) to model the power spectrum C gg (k). Fourier-transforming the above equations gives the halo model predic- 
tions for the projected cross-correlation ii; cross to compare with the measurement. We will use the above equations to derive the averaged 
halo parameters and off-centering parameters in the following analysis. 



M=M 

(zlrg) 



3 MEASUREMENTS AND RESULTS 
3.1 SDSS DR7 LRG Catalog 

Here we describe the data used for the analysis in this paper, all of which come from the Sloan Digital Sky Survey Data Release 7 catalog 
(SDSS DR7). The SDSS (York et al. 2000a) imaged roughly n steradians of the sky, and followed up approximately one million of the 
detected objects spectroscopically (Eisenstein et al. 2001a; Richards et al. 2002; Strauss et al. 2002). The imaging was carried out by drift- 
scanning the sky in photometric conditions (Hogg et al. 2001; Ivezic et al. 2004), in five bands (ugriz; Fukugita et al. 1996; Smith et al. 
2002) using a specially-designed wide-field camera (Gunn et al. 1998). These imaging data were used to create star and galaxy catalogs that 
we use in this paper. All of the data were processed by completely automated pipelines that detect and measure photometric properties of 
objects, and astrometrically calibrate the data (Lupton et al. 2001; Pier et al. 2003; Tucker et al. 2006). The SDSS I/II imaging surveys were 
completed with a seventh data release (Abazajian et al. 2009a), though this work will rely as well on an improved data reduction pipeline 
that was part of the eighth data release, from SDSS III (Aihara et al. 2011); and an improved photometric calibration ('ubercalibration', 
Padmanabhan et al. 2008). 

We use the SDSS DR7 LRGs in the Northern Galactic Cap with a contiguous region, which is made publicly available by Kazin et al. 
(2010). The sample consists of 96,762 LRGs with absolute magnitude —23.2 < M g < —21.2 in the redshift range 0.16 < z < 0.47covering 
about 1.58 (Gpc/fe) 3 comoving volume in the concordance ACDM model, with nearly constant comoving number density for z < 0.36 and 
decreasing number density above that redshift. To recover the halo density field from the LRG distribution, we used the Counts-in-Cylinders 
(CiC) techniques developed by Reid & Spergel (2009) to identify groups of A^lrg ^ 2 LRGs occupying the same halo. To be more precise, 
multiple LRGs are considered as neighbors of the same group when the transverse separation satisfies Ar± ^ 0.8/i _1 Mpc and the redshift 
difference satisfies Az/(1 + z) ^ 0.006 corresponding to Sv p = 1800 km/s. The radial-direction length of the cylinder is motivated by the 
virial motion of galaxies in a massive halo, and was shown to safely include satellite galaxies in the same halo using mock catalogs (Reid 
et al. 2009). Then LRGs in different CiC groups are grouped together by a Friends-of-Friends method (by connecting any of the member 
galaxies in the different CiC cylinders). Table 1 summarizes the reconstructed LRG groups. The total number of halos inferred from the LRG 
catalog is 92,046, and about 4.5 per cent are multiple-LRG systems. Among the multiple-LRG systems, about 90 per cent are systems with 
two LRGs (TVlrg = 2). We also generate random catalogs that have the same area coverage and redshift distribution as the halos; these are 
used to estimate the LRG-galaxy lensing signal and the cross-correlation with photometric galaxies in the imaging data. 

To compute the power spectrum and the cross-correlations from the LRG-inferred halos, we must define the halo center for each LRG 
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Number of LRG(s) (-/Vlrg) m eacn FoF group Number of LRG FoF groups (fraction) 

1 87889 (95.5 per cent) 

2 3713 

3 358 

4 65 

5 14 

6 6 

7 1 

Total 92046 (100 per cent) 



Table 1. The number of different LRG groups after the CiC-FoF based group finder of the SDSS DR7 LRG distribution. "-/Vlrg = 1" means single LRG 
systems, where each region has a single LRG in the cylinder region in angular and redshift space. "./Vlrg -2 2" systems are multiple-LRG systems, where 
each region contains -/Vlrg LRGs in the cylinder region. 
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Figure 1. The histogram of magnitude differences between the brightest and faintest LRGs (BLRG and FLRG) in multiple-LRG systems (see Table 1). The 
solid-line histogram is for the systems having 2 LRGs inside, while the dashed line is for the systems with -/Vlrg ^ 3. 



system. (1) For single LRG systems with -/Vlrg = 1, we use the redshift and angular position of LRG as the halo center proxy. (2) For 
multiple-LRG systems with /Vlrg ^ 2, we use the following three halo center proxies in order to compare the measurements: 

• BLRG: the brightest LRG (BLRG) in each multiple-LRG group as the halo center. The BLRG is expected to reside in the most massive 
sub-halo and therefore be closer to the underlying true center due to dynamical friction theory. 

• FLRG: the faintest LRG (FLRG) as the halo center; it can be considered as the extreme counterpart to BLRG. 

• Mean: the mean position of LRGs in their redshift and angular positions. Note that we did not use any luminosity or other weighting to 
estimate the mean position. This is the halo-center proxy used in Reid et al. (2010). 

Fig. 1 shows the distribution of magnitude difference between the brightest and faintest LRGs (BLRG and FLRG) in the multiple-LRG 
systems. The magnitude difference is typically within 1 mag. Most of the 2-LRG systems have a magnitude difference that is less than a few 
tenths of a magnitude, implying that the BLRG and FLRG are not very different, and the FLRG may be closer to the true center than the 
BLRG in some halos, as we will address below. 



3.2 LRG power spectrum 

We first use the angle-averaged, redshift-space power spectrum of LRG-inferred halos to study the effect of off-centered LRGs. We employed 
the method of Reid et al. (2010) to measure the power spectrum from the reconstructed halo density field, as briefly summarized below. 

The power spectrum measurement was done using the Fourier decomposition method developed in Feldman et al. (1994). We used 
1024 3 grids in a cubic box with side length 2600 Mpc//i, which covers the entire region of the LRG distribution in redshift space. The 
density fluctuation field of halos on a grid is estimated by placing the halos onto the grid and subtracting an unclustered "random" catalog, 
which matches the halo selection. The random catalog is given from Kazin et al. (2010), where the variations in angular and radial selections 
and the effect of fiber collisions are taken into account. In doing the density assignment from the LRG-halos or random catalogs, we used the 
weighting method in Percival et al. (2004) using a luminosity-dependent bias model. However, note that we used the luminosity of BLRG in 
each halo to compute the weighting, and used the same weight for all the three BLRG, FLRG and Mean centers in the multiple-LRG systems. 
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Figure 2. The angle-averaged redshift-space power spectra for the LRG-inferred halos, measured using the different halo center proxies for the multiple-LRG 
systems: the brightest LRG position (BLRG), the faintest LRG position (FLRG) and the mean position (Mean) of LRG positions in angular and redshift space 
(see Section 3.1 for the details). The circle and triangle symbols show the fractional differences of the halo power spectra of the BLRG or FLRG centers, 
respectively, relative to the spectrum of the Mean center. Since we used the same halos and power spectrum measurement methods, the difference arises 
solely from the different halo centers of the multiple-LRG systems. The FLRG power spectrum shows a larger suppression compared to the other two, which 
indicates that the FLRGs have the largest offsets and thus largest internal velocity dispersion in the multiple-LRG systems. The BLRG power spectrum still 
shows some suppression compared to that for the Mean, which indicates a residual off-centered effect of BLRGs. The solid and dashed curves in each panel 
show the best-fit FoG model assuming the Lorentzian (left panel) and Gaussian (right) functional forms. Note that the measurement points are the same in 
both panels. 



Thus we used the same random catalogs and the same luminosity weights for all the three BLRG-, FLRG- and Mean-center catalogs in order 
to avoid any unwanted effect on the measured power spectra. We also note that, in the following results, we focus on relative differences 
between the measured power spectra and cross-correlation functions using the different centers, where the differences should not be sensitive 
to any details of the measurement methods. 

To estimate the error bars or measurement uncertainties, we divide the LRG sample into 100 equal-area, nearly contiguous regions 
on the sky, and carry out the measurements of the LRG-inferred power spectra from each of those 100 regions (Mandelbaum et al. 2012b, 
hereafter Ml 2); for some description of the limitations of bootstrap and jackknife resampling, see, e.g., Hartlap et al. (2007) and Hirata et al. 
(2004). Then we estimate the covariance matrix from the 100 power spectra. The jackknife resampling assumes independence of the regions, 
and thus cannot be used to obtain the error bars for scales comparable to the size of the regions (which are far larger than the maximum scales 
used for our fits). 

Fig. 2 shows the measured power spectra of halos reconstructed from the SDSS LRG catalog. Plotted here is the fractional difference of 
the power spectra, measured by using either BLRG- or FLRG-halo centers, relative to the power spectrum calculated using the Mean-center 
in the multiple-LRG systems. The error bars at each k bin are estimated from the jackknife method we described above. Note that, in this 
analysis, we changed the redshift- and angular positions only for 4.5 per cent of the halos (the other 95.5 per cent are exactly the same 
since they include only a single LRG). The figure clearly shows that the power spectra calculated using both the BLRG and FLRG centers 
show smaller amplitudes with increasing k than the power spectrum calculated using the Mean center, implying that the BLRG and FLRG 
results are more affected by the FoG effect, leading to a stronger suppression in the power than the Mean spectrum. The significance of the 
differences is A\ 2 = 69, 120 and 138 for the BLRG, FLRG and the combined BLRG and FLRG difference, respectively, corresponding 
to about 8.2, 10.9, and 11.7cr detections of the FoG effects, respectively. In these estimates, we included the correlations between the error 
bars at different k bins (which are typically around 0.1, but occasionally as large as 0.5), and the (significant) correlation between the BLRG 
and FLRG spectra using the full jackknife covariance matrix. The FLRG spectrum has more suppressed power, which is expected since 
fainter galaxies are more likely to be satellites, but it is more surprising that the BLRG power spectrum also shows the FoG suppression. This 
suggests that the BLRGs also have an off-centered distribution in the host halos, or some of the BLRGs are satellite galaxies. On the other 
hand, the power spectrum of the Mean center turns out to be least affected by the FoG effect, which is counter-intuitive, because there is no 
LRG at the inferred position. However, we note that most of the multiple-LRG systems have A^lrg = 2 and a small magnitude difference 
between the two LRGs. Hence, some FLRGs, rather than the BLRGs, may be central galaxies in some of the host halos. 
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Following the method developed in Hikage et al. (2012), we can make a model interpretation of the measured power spectra for the 
different LRG-inferred halo centers. The redshift-space power spectrum we measure arises from auto- and cross-correlations of/between the 
single- and multiple-LRG halos, where the fraction of multiple-LRG halos among all the LRG-inferred halos is only 4.5 per cent, which 
we hereafter refer as /m-lrg = 0.045. We assume that only some fraction of the chosen LRGs are central galaxies in the single- or 

o LR.G Ivl LRG BLRXT 1 FLR.C* 

multiple-LRG systems, and represent these fractions as q ccn or q ccn (to be more precise, we will consider q ccn * or q ccn for the 
multiple-LRG halos, because we chose the galaxies for the halo center proxies). The remaining galaxies, given by (1 — (fcen), are satellites 
that cause the FoG effect. Including the parameter q C cn gives a better agreement with measurements of the LRG-galaxy lensing and the 
projected cross-correlation function of LRG-halos and photo-z galaxies, to be described in subsequent sections. Thus, we can model the 
redshift-space power spectrum of the LRG-inferred halos as 



Ps{k,fl) = [l + /J^l 2 ] 2 &S-LRG(1 - /m-LRg) j<7ccn LRG + (1 - 9con LRG )V F S-LRG(fc,M)| 

+&M-LRg/m-LRG |<7con + (1 — <£ln LRG )\/ ^M-LRG M) | 



P^(k), (15) 



where &s-lrg and &m-lrg are the average bias parameters for the single- and multiple-LRG host halos, respectively; q^ 
of central LRGs in the single-LRG systems; q^l~ LRG is the fraction of either BLRGs or FLRGs to be centrals in the multiple-LRG systems. 
We have assumed that the redshift-space distortion effect due to the peculiar velocity field of halos is given by the Kaiser formula (Kaiser 
1987), given by the function [1 + Pf-i} 2 , and /3 = (l/fe)dln D/d In a (D is the linear growth rate). P^ L (k) is the real-space, nonlinear power 
spectrum of dark matter, and F(k,fi) is the velocity function to model the average velocity distribution of off-centered (satellite) LRGs 
within the host halos (see below). The coefficients (1 — /m-lrg) and /m-lrg are the fractions of the single- and multiple-LRG halos 
relative to all the LRG-host halos, respectively. In the above equation, we assumed that the central LRG rests at the center of the host halo 
and therefore has no FoG effect. Thus, different populations of LRG-host halos (the single- and multiple-LRG systems in our case) cause 
FoG effects on the redshift-space power spectrum in different ways, varying with their fractions, the halo bias parameters and the fractions 
of satellite LRGs in each population. To compute the bias parameters, for simplicity we employ the halo bias using the average halo masses 
measured from the LRG-galaxy weak lensing for the assumed fiducial cosmology: 6s-lrg = 2.12 and &m-lrg = 3.26 for the average 
masses of the host halos, Ms-lrg = 0.4 x 10 14 Mq I h and Mm —lrg = 1-6 x 10 14 M©//i, respectively, estimated using the bias model 
in Sheth & Tormen (1999) 13 . For the velocity function F, we assume the following functional forms that are often used in the literature 
(Hamilton 1998): 

i 

(Lorentzian FoG effect) 



F(k,n)={ I 1 + (kfia v , a/aH(z))2]' (16) 
exp[— (kfi(T V! oB /aH(z)) 2 ], (Gaussian FoG effect) 

where a v , B is the typical velocity dispersion of off-centered LRGs. Note that "^A" of ■s/F in the FoG effect of Eq. (15) was introduced 
because the above functions are conventionally used to model the FoG effect on correlations between two satellite galaxies. Thus Eq. (15) is 
different from the form used in the literature, and predicts a non-trivial scale dependence as a function of wavenumber; e.g., at the large-fc 
limit, the FoG function asymptotically approaches a constant value, [6s-lrg(1 — /m-lrg )g? C n LRG + ^m-lrg/m-lrg?^ lrgj2^ w jjjj e 
the standard FoG function F — > 0. 

The fractional power spectra shown in Fig. 2 are with respect to the power spectrum for the "Mean" centers in the multiple-LRG 
systems. Hence we need to model the power spectrum for the Mean center based on our method. Fig. 2 implies that the Mean centers, the 
mean position of BLRG and FLRG in multiple-LRG systems, are statistically closer to the true centers, such that the power spectrum is least 
affected by the FoG effect. Physically, this is very difficult to imagine, since we expect that the Mean center, which has no corresponding 
galaxy at the position, should be offset from the halo center to some degree. To consider the offset probability of the Mean centers, we need 
to include three cases: (1) within the host halo, BLRG is a central galaxy, while FLRG is a satellite galaxy, (2) FLRG is central, while BLRG 
is satellite, and (3) both BLRG and FLRG are satellites in their host halo. Among all the multiple-LRG systems, the probabilities of these 3 
cases are given by qf^ G , gf C n RG and 1 — g^n RG — 9con RG > respectively. With these in mind, the velocity dispersion of the Mean center can 
be estimated as 

I FLRG\ 2 (n- BLRG \ 2 ( rr BIjRG Y 2 4- ( rr FLRG \ 2 

I Mean\ 2 BLRG \"v.oS ) FLRG \"v.oS ) I _ BLRG _ FLRG\ \"v,oB ) T \"y,off ) 

(^u,off J Qccn a * Qccn ^ V Qccn Qccn J a 

(1 - Q^°) K L o RG ) 2 + (1 - 9ccn RG ) (< L JP) 2 

4 ' ( ' 

where the three terms in the first line correspond to the three cases mentioned above, and we have assumed no correlation between the 



3 While we have used a theoretical bias vs. halo mass relation, it is worth noting that the large-scale biases for LRG samples that together compose the LRG 
sample in this work were found (via direct-fitting to large-scale clustering and lensing measurements) to be 2.07 ± 0.05 and 2.26 ± 0.06 (Mandelbaum et al. 
2012b). Thus we expect that the average of bs— LRG an d ^M— LRG (weighted by the relative fractions of single- and multiple-LRG systems) to lie in between 
those two measurements, which is indeed the case given the predominance of single-LRG systems. This confirms that use of these theoretical predictions is 
consistent with the data. 
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distributions of BLRG and FLRG in each host halo as well as no correlation between the distributions of LRGs in different halos. We also 
assumed that BLRG and FLRG determine the mean position of LRGs in multiple-LRG systems, which may host more than two LRGs 
(-/Vlrg ^ 3). Or equivalently we assumed that the Mean centers are determined mainly by the multiple-LRG systems with 2 LRGs, because 
the systems with 2 LRGs are a dominant population of the multiple-LRG systems (see Table 1). Thus we expect that the Mean centers 
always have offsets from the true center. The factor of 4 in the denominator of each term is from the fact that we computed the dispersion of 
the off-centering amount of the Mean centers. For instance, in the first case (the first term in the above equation), suppose the line-of-sight 
displacement vector for the Mean center with respect to the true center in each system is defined as friiMean = ^ r ||,FLRG/2 as the BLRG 
is at the halo center ($7"| ,blrg — 0). Then the average (tfr^Mean) = and the dispersion (u^g an ) 2 = ((5r|| iMoan ) 2 ^ = (cr^g RG ) 2 /4, 
yielding the factor of 4. 

Hence, the fractional power spectrum shown in Fig. 2 can be modeled based on our method, e.g. for the BLRG center, to first order in 

/m-lrg: 



1 d/i 



-f( 1 + A" 



2x2 



+ (i - gc B c L n RG ) v /F(fc, M ;< L | G ) - ^(fc,M;< T 



BLRG 
Qccn 

( k ) ( &M-LRG \ J-l 1 L ' ' ^ ,.„. 

1 ~ 2/m-lrg -f I 71 ■ (lo) 



■ v lrg j £ d i(i + ^r 

Thus the quantities such as P NL (fc) (or f3 to a good approximation) cancel out in the fractional difference, and the ratio of bias parameters 
(&m-lrg/&s-lrg) is relevant. In the above equation, we also assumed that the FoG effect for the single-LRG systems is negligible, because 
the halos have a higher fraction of the central LRGs (gccn LRG — 0.8) and a smaller velocity dispersion due to the smaller halo mass as we 
will show in the following. Since f^off 1 ls given in terms of cr^o RG and (and we have fixed the background cosmological model to 

compute the linear Kaiser factor /3), the relevant fitting parameters are 4: g^n RG > gf C n RG , °" R o RG ' an d a v off G > respectively. 

The solid and dotted curves in the left- and right-panels of Fig. 2 show the best-fit FoG effect for the Lorentzian or Gaussian FoG 
models, respectively. In this fitting, for simplicity, we assumed that the fraction of central BLRGs or central FLRGs is g R n RG = 0.54 or 
9ccn RG = 0.32, respectively, as implied from the measurements of the LRG-shear lensing and the LRG-photo-z galaxy cross-correlation in 
the subsequent sections. We find that the best-fit velocity dispersions for the satellite BLRGs and FLRGs are o" R off G ~ 519 ± 48 km/s and 
<?f,o RG = 561 ± 37 km/s for the Lorentzian FoG effect, and a B ^ RG = 498 ± 41 km/s and cr^ofP = 512 ± 32 km/s for the Gaussian 
FoG effect, respectively. For the Mean center, <7^X ~ 291 ± 20 km/s and o^X - 270 ± 17 km/s for the Lorentzian and Gaussian FoG 
models, respectively. Fig. 2 shows that the Gaussian FoG model gives a better fit to the scale dependence of the FoG effect measured from 
the SDSS LRG catalog than the Lorentzian model. The minimum \ 2 values inferred are for the best-fit models obtained by jointly fitting the 
model predictions to the BLRG and FLRG power spectra taking into account the cross-covariance. The \ 2 has fourteen degrees of freedom: 
sixteen data points minus two model parameters (two a Vt0 g parameters). These results are confirmed by the results using the LRG-galaxy 
weak lensing and the LRG-photo-z galaxy cross-correlation. 



3.3 LRG-galaxy weak lensing 

As the second observable, we use the cross-correlation of LRG-inferred centers with shapes of background galaxies - the so-called galaxy- 
galaxy lensing using the LRGs as lenses, which was measured in the same way as in Ml 2. In brief, this measurement uses a catalog of 
1.2 background galaxies per arcmin 2 with measured shapes from the re-Gaussianization method (Hirata & Seljak 2003) and photometric 
redshifts from Zurich Extragalactic Bayesian Redshift Analyzer (ZEBRA, Feldmann et al. 2006); the catalog was described and thorough 
studies of systematic errors were carried out in several recent papers (M12, and also Reyes et al. 201 1; Mandelbaum et al. 2012a; Nakajima 
et al. 2012). 

The lensing measurement is done as follows. We first identify background galaxies around each LRG that have photometric redshift 
larger than the LRG spectroscopic redshift. The weights are assigned to each LRG-background galaxy pair using optimal weighting according 
to the error on the source shape measurement via 

«*« = S 2 (J,„2 V (19) 

where a 2 is the shape measurement error due to pixel noise, and a 2 SN is the RMS intrinsic ellipticity (both quantities are per component, rather 
than total; the latter is fixed to 0.365 in M12. £ C rit is the critical surface mass density denned by H~\ t (z h z s ) = AtyGc' 2 D u Di{l+z{f / D B , 
where Di and D s are the angular diameter distances to lens (LRG) and source and Di s is the distance between them, and we used the best-fit 
photo- z of each source to compute D s ; this gives rise to a bias in the signals that can be easily corrected using the method from Nakajima 
et al. (2012). The factor (1 + zi)' 2 arises due to our use of comoving coordinates. The factor S~ 2 t downweights pairs that are close in redshift 
and therefore are inefficient in weak lensing. 

The projected mass density in each radial bin can be computed via a summation over lens-source pairs "Is" and random lens-source 
pairs "rs": 
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Figure 3. Left panel: The cross-correlation of LRG-inferred halos with shapes of background, photometric galaxies - the LRG-galaxy weak lensing, measured 
for the multiple-LRG systems using the BLRG, FLRG or Mean halo center proxies, respectively. The background galaxy sample consists of photometric 
galaxies with photometric redshifts larger than the LRG redshift (see text for the details). In these lensing measurements, we used the exactly same lens-source 
pairs and random point catalogs. The three lensing signals at radii > 1 Mpc/h are similar, but the signals at smaller radii differ from each other. Right panel: 
Likewise, the projected cross-correlation function of the LRG-inferred halos with photometric galaxies for the multiple-LRG systems, using the three different 
centers. The photometric galaxy sample is taken from the photometric galaxies for which the photometric redshift is consistent with the spectroscopic redshift 
of each LRG halo within the photo-^ errors. The measured amplitudes at scales R ^ 0.2 Mpc//i show increasing off-centering effects in the order of the 
Mean, BLRG and FLRG centers as in the FoG results in Fig. 2 (see text for details). The BLRG correlation shows an excess in the power at the small radii, 
R £ 200kpc//i. These cross-correlations of different centers can be interpreted by using models with a mixture of centered and off-centered (satellite) LRGs 
in the host halos (see text for discussion). 



AE(R) = W - , (20) 

where et is the tangential ellipticity component of source galaxy with respect to the lens position, the factor of 27?. is necessary to convert to 
tangential shear 74 due to our definition of ellipticity, and R is the comoving projected radius from lens, which is estimated from the separation 
angle between lens (LRG) and source on the sky as R = xlrg#- The division by ^2 w rs is necessary to account for the fact that some of 
our 'sources' are physically associated with the lens, and therefore not lensed by it. Finally, we subtract off a similar signal measured around 
random lenses, to subtract off any coherent systematic shear contributions (Mandelbaum et al. 2005); this signal is statistically consistent 
with zero for all scales used in this work. 

To calculate the error bars, we used the jackknife resampling method, as for the power spectrum measurement. The maximum scale 
used for the fits in the lensing analysis is a few Mpc, within a factor of two of the virial radius of massive halos; this corresponds to several 
arcminutes at the typical LRG redshift, which is < 0.01 times the typical size of each jackknife resampled region. Thus, the jackknife method 
is a reasonable approach to getting the covariance matrix for the projected mass profile. 

The left panel of Fig. 3 shows the LRG-shear cross-correlation measured for the multiple-LRG systems, using the different halo centers 
(Mean, BLRG and FLRG). We used the same pairs of lens and sources, and therefore the difference in the measured correlations arises from 
the different centers. We are plotting only the radii relevant for the virial region of massive halos, up to a few Mpc. The three mass profiles 
agree with each other at radii > 1 Mpc//i, reflecting that the three centers are in the same halos. However, the mass profiles differ at smaller 
radii. The intermediate scales around 0.5 Mpc/h show that the lensing signals for the BLRG and Mean centers are higher than the FLRG 
signal, consistent with our interpretation of the FoG measurements in Fig. 2. At the smaller scales, the mass profile for the Mean center shows 
a decreasing power with decreasing radii, which is a clear signature of the dilution effect due to the off-centering effect and the lack of a 
galaxy at that position. On the other hand, the mass profiles of the BLRG and FLRG centers do not show this dilution at the smallest scales. 

In Fig. 4, we show the results of fitting the halo model prediction from Section 2.3 to the measured lensing profiles for the three centers 
in the left panel of Fig. 3. The fit includes 10 model parameters 

/ - BLRG D BLRG BLRG FLRG D FLRG FLRG D Mean /n , . 

Pa — (M18O6, C1806, 5ccn l-Kofl ! m sh ,9con ) -"-off , m sh , -"-off , °) , (21) 

where Misot and Cigo;, are the average mass and concentration parameter of NFW profile for host halos of the multiple-LRG systems, g cen 
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Figure 4. The best-fit halo model predictions, developed in Section 2.3, to the measured LRG-galaxy lensing signals, for the the BLRG (left panel), FLRG 
(middle) and Mean (right) centers, respectively. The lower panel in each plot shows the ratio of the best-fit model to the measured profile. The data with error 
bars are the same as in the left panel of Fig. 3. We considered the lensing contributions from the host halos of the multiple-LRG systems and from the sub-halo 
hosting the BLRG or FLRG at small scales. For the host halo contribution, we considered 3 contributions: the 2-halo term (dashed curve) and the 1-halo terms 
with and without the off-centering effect, indicated by the long- and short-dashed curves, respectively. For the BLRG- and FLRG-centers, we assumed that 
some fraction of the BLRGs or FLRGs are central galaxies, modeled by q con (see Eq. 21), while the remaining LRGs are off-centered (satellite) LRGs. On the 
other hand, we assumed that all the Mean center have offsets from the true center in each host halo. The dot-dashed curve is the sub-halo lensing contribution 
for which we assumed a point mass. The top solid curve in each panel is the total lensing signal including all contributing terms. Note that the mass and 
concentration parameters of the host halos are the same in the three panels, since the same set of halos was used in each case (we simultaneously fit the model 
to the three measurements). 
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Table 2. The best-fit values and marginalized errors of the model parameters obtained from the fitting of the models to measurement of the LRG-galaxy weak 
lensing or, separately, the projected cross-correlation of LRGs and photo-2 galaxies (Fig. 3). We focused on the multiple-LRG systems for the measurements, 
and the error bar of each parameter is the 1-cr uncertainty including marginalization over other parameters. 



is the fraction of the BLRGs or FLRGs that are central galaxies in their host halos, R a g is the typical off-centering radius (see Eq. 1), m sn 
is the sub-halo mass hosting the BLRG or FLRG, and b is the bias parameter for the 2-halo term of the projected mass profile (Eq. 9). We 
do not assume a fixed value for the halo bias, instead treating b as a free parameter to absorb any uncertainties in the modeling of this term 
(e.g., linear vs. nonlinear power spectrum, linear bias only, etc.). This is acceptable because even at the maximum scales shown on the plot, 
the 2-halo term is subdominant to the 1-halo term. We assumed a narrow mass range of the host halos. We also assume that all the "Mean 
centers" in all the multiple-LRG systems are off-centered (i.e. g,^ n an = 0) and lack a sub-halo lensing contribution, because the Mean center 
is not associated with the position of any bright galaxy. Notice that we used the average halo mass parameter, defined as the mass enclosed 
within a sphere of radius risob inside of which the mean density is 180 times the mean background mass density pm,o- Since we are working 
on the same set of multiple-LRG systems, we use the same parameters of the host halo, Mi8o& and cisob, when jointly fitting the models 
to the measured mass profiles of the three centers. We used the Markov Chain Monte Carlo (MCMC) method to explore the marginalized 
posterior distribution of each model parameter, by taking into account the significant correlations between the observed lensing profiles using 
different halo centers, and the different radial bins. 

Table 2 summarizes the parameter constraints derived from the measured LRG-galaxy weak lensing. First of all, the host halo mass 
for the multiple-LRG systems is about 1.6 x 10 14 Mq//i, which is significantly more massive than the average halo mass inferred from the 
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LRG-shear lensing using all the LRG systems, about 4 x 10 M@/h (Mandelbaum et al. 2006a) as explicitly studied in Fig. 8. Secondly, 
the central galaxy fraction for the BLRGs is qf^ G — 63 per cent, and larger than 24 per cent for the FLRGs. The measured mass profiles 
favor non-zero g con values at the 2 - 3a level, i.e. some contribution from central LRGs is needed to reproduce the measured profiles. We 
found non-zero values of the off-centering radius for all three different centers (BLRGs, FLRGs and Mean): R a ~ 350, 400 and 190 kpdh 
for the BLRG, FLRG and Mean centers, respectively, although this detection is not very statistically significant for the BLRG center (2a). 

The different panels of Fig. 4 compare the best-fit model predictions with the measured mass profiles for the three different centers. It is 
clear that our model reproduces the measurements very well. The mass profile for the Mean center has a clear signature of dilution due to the 
off-centering effect. This is reasonable, because we do not believe that the Mean position of LRGs happens to coincide with the true center. 
For the lensing mass profiles for the BLRG and FLRG centers, the best-fit model has two contributions from the central LRGs and from the 
off-centered (satellite) LRGs. The sub-halo contribution of the BLRG or FLRG is a marginal detection, but implies that the measured lensing 
profile needs such an additional mass concentration relative to the smooth NFW profile, which can be caused by two effects: the sub-halo 
hosting the satellite LRGs, or the baryonic concentration effect on the total mass around the central LRGs (Schulz et al. 2010). The sub-halo 
mass we measured could be due to these effects and therefore is difficult to interpret physically; furthermore, the detection significance is 
quite low, < 2a for both BLRG and FLRG. 

Since we now have constraints on the mass profile of host halos and the typical off-centering radius for the different halo centers, we 
can infer the typical velocity dispersion of the off-centered LRGs based on the virial theorem (Hikage et al. 2012): 

a , > GM(< r) 

<7 a,v{r) = i ■ (22) 

2r 

The enclosed mass within a sphere of radius i? ff for an NFW halo is given as M(< R) — Misobf(cisobR/Risob)/f{cisob), where 
f(x) = ln(l + x) — x/(l + x) and Misoi,, -Ri80i> and Cisot are the virial mass, virial radius and the halo concentration for the mass 
definition with respect to 180 times the mean mass density. As can be found from Table 2, if we assume M 1S0b = 1.6 x 10 14 M o /h, 
Cisob — 4.8, and the offset radii of R g = 440 and 390 kpc/h for the off-centered BLRGs and FLRGs, the virial theorem above gives 
er^off ~ 516 and 511 km/s, respectively. To be more precise, we estimate the typical velocity dispersion for the off-centered LRGs by 
weighting the velocity dispersion in the above equation with the off-centering profile with width of R g (Eq. 1). These velocity dispersions 
are consistent with those inferred from the FoG measurements in Fig. 2 (which imply a v , a g ~ 500 km/s). 

3.4 Projected cross-correlation between LRGs and photometric galaxies 

As the third observable to constrain the off-centered LRGs, we use the projected cross-correlation function between the LRG-inferred halos 
and a purely photometric sample of galaxies, w CTOSS (R). The sample used for this calculation was selected in much the same way as the 
lensing source catalog described in the previous section, but (1) there is no requirement that the galaxy be large enough in angular extent to 
measure a shape, (2) we impose a brighter flux limit of r < 21 to reduce the effects of photo-z error, and (3) we require that the ZEBRA fit 
to the galaxy SED be consistent with an early-type galaxy and that the photo-z is consistent with the LRG redshift to within the quoted 68 
per cent statistical errors from ZEBRA. 

Our measurement of the projected correlation between the LRG-inferred halos (H) and fainter photometric red galaxies (G) also makes 
use of a sample of random points (R) distributed in the same way as the halos. Given a lower and upper 68 per cent confidence limit on the 
faint galaxy photo-z, z\\ and z u i, we compute the projected correlation via summation: 

cross , m HG(Z U Z H < ggl) 

where the projected radius is estimated from the redshift of the LRG halo and the angular separation between the halo center and the 
photometric galaxy, R = xlrg The cross-correlation method is very powerful to distinguish the photometric galaxies that are physically 
associated and therefore clustered with the spectroscopic LRG halos. Including photometric galaxies that are not associated with LRGs, and 
only appear at that redshift due to photo-z error, causes a dilution of the correlation signals. Hence, we will not use the amplitude information 
for the following parameter estimation. Similarly to the LRG-galaxy lensing measurement, we estimate the covariance matrix of TO crosB 
via the jackknife resampling method. 

The right panel of Fig. 3 shows the measured cross-correlation, w CTOBB (R), for the three halo centers. Again we worked on the same set 
of multiple-LRG systems for the three centers, and the difference between the three results is only due to the positions of the LRG-inferred 
halo centers. Interestingly, over the intermediate range of radii, R ~ [0.15, 0.4] Mpc/h, the correlation functions have greater amplitudes in 
the order of the Mean, BLRG and FLRG centers. Then, for the larger radii, R > 0.4 Mpc/h, the correlation amplitudes are in the opposite 
order, greater in the order of the FLRG, BLRG and Mean centers 14 . These behaviors are exactly what is expected for the off-centering effects, 
because the off-centering effect causes a transfer of the power of the correlation function from smaller to larger radii relative to the typical 
off-centering radius. The order of the off-centering amounts for the BLRG, FLRG and Mean centers is in good agreement with the FoG 

14 While the magnitude of this effect appears small within the errors on the curves, the signals are highly correlated such that the statistical significance of 
these orderings is high. 
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Figure 5. The best-fit model predictions obtained by fitting the models (see Section 2.4) to the projected cross-correlation functions of the LRG-inferred 
halos with the photometric galaxies, for the BLRG (left panel), FLRG (middle), and Mean (right) centers, respectively. As in Fig. 4, we assumed that only 
some fraction of the BLRGs or FLRGs are off-centered galaxies. By using a model containing a mixture of central and satellite LRGs, we can reproduce the 
measured projected correlation functions over the entire range of radii we consider. The lower plot in each panel shows a remarkably good agreement between 
the best-fit model and the measured profile for each center. 



measurements in Fig. 2, and also consistent with the weak lensing measurements in the left panel of Fig. 3. On the other hand, the correlation 
function for the BLRG center shows the greatest amplitude at the very small radii, R < 0.15 Mpc/h, which is not as expected from the 
off-centering effect. We will show below that the measurements can be nicely explained by including a mixture of central and satellite LRGs 
in the multiple-LRG systems into the halo model predictions. 

Similarly to the LRG-galaxy lensing, we fit the halo model in Section 2.4 to the projected correlation functions. To be more precise, we 
use the following model parameters: 

/ a j\> - BLRG D BLRG FLRG D FLRG D Mean r2x 

Pa — (ZAZphz, Misob, ClSOb, <?cen J -"off ! 9cen > "off ) -"-off !° )■ \ Z ^> 

Most of the parameters are the same as those for the LRG-galaxy lensing analysis (Eq. 21), except for the parameter Az p i lz and b 2 . The 
parameter Az p i lz is a nuisance parameter to model the dilution factor, ^^(^lrg)/^^ all , that arises from a contamination of unassociated 
photometric galaxies due to photo- z errors as we discussed above. We also include the bias parameter b 2 to account for an uncertainty in the 
2-halo term amplitude (which is for ^LRG^phg in Eq. 14). In our fitting, we assumed a single (i.e. very narrow) mass bin for host halos of the 
multiple-LRG systems, and that the distribution of photometric galaxies in the halos follows an NFW profile. 

Table 2 shows the best-fit value and the marginalized la errors for each parameter. The results are in remarkably good agreement with 
the LRG-galaxy lensing results within the error bars, while the two measurements are independent in a sense that they are subject to largely 
different systematic errors, and used different catalogs of photometric galaxies: the background galaxies behind the LRGs and the galaxies 
at the same redshift as the LRGs, respectively. 

Fig. 5 shows the best-fit model predictions compared with the measurements. The halo model can well reproduce the measured correla- 
tion functions over all the range of radii we consider, including the enhancement in the correlation amplitude at the small radii for the BLRG 
center. 

In Fig. 6, we show the marginalized constraint contours in each two-dimensional sub-space of the off-centering parameters. Comparing 
the dotted, dashed and solid contours shows how the parameter degeneracies are broken by changing the smallest radius down to which the 
cross-correlation signals are included in the model fitting. It is clear that the enhancement in the cross-correlation amplitudes at small radius, 
R 0.3 Mpc//i, helps to break parameter degeneracies, especially to constrain the central galaxy fraction parameters g C en- 

Fig. 7 compares the constraint contours obtained from the model fitting to the LRG-galaxy weak lensing and the LRG-photometric 
galaxy cross-correlations. The figure shows that the two results are in good agreement with each other within the errors, and that the cross- 
correlation gives tighter constraints on the off-center parameters than the weak lensing due to the higher signal-to-noise ratios. The difference 
in S/N between the two types of measurements will likely be different in deeper upcoming lensing surveys than it is in SDSS. 
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Figure 6. The mai'ginalized constraint contours (68 and 95 per cent C.L.) in two-dimensional sub-spaces of the off-centering parameters. The dotted, dashed 
and solid contours in each panel show the results when we change the minimum radius for the fit to the cross-correlation functions of LRGs and photometric 
galaxies in Fig. 5, i?. m i n = 0.3, 0.2 and 0.1 Mpc/h. Including the smaller-radius cross-correlation helps to break the parameter degeneracies, especially the 
fraction of central LRGs, g C en. 



4 IMPLICATION FOR THE RESIDUAL FOG EFFECT IN THE LRG POWER SPECTRUM 

We have so far shown that three different measurements (the angle-averaged redshift-space power spectrum, the LRG-shear correlation 
function and the cross-correlation of LRGs and photometric galaxies), all imply the existence of satellite (off-centered) LRGs for the multiple- 
LRG systems, which represent ~ 5 per cent of the halos hosting LRGs. Perhaps most surprising is that some of BLRGs in the multiple- 
LRG systems exhibit this off-centering effect. In this section, based on these results, we discuss the implications for possible residual FoG 
contamination of the LRG power spectrum. For some of the multiple-LRG systems, the BLRG is brighter than the FLRG only by a few tenths 
of a magnitude (see Fig. 1). One might think that the off-centered BLRGs are mainly from the multiple-LRG systems where the BLRG and 
FLRG have a small magnitude difference and therefore the two LRGs are not different (the FLRGs are central instead). In Appendix A, 
by dividing the multiple-LRG systems into halved samples which have the larger and smaller magnitude difference between the BLRG 
and FLRG than the median, we study whether or not the amount of the off-centered BLRGs differs in between the halved samples. We do 
not find any significant difference for the off-centered BLRGs in the two samples, although the FLRG center shows a significant difference. 
Therefore, the current data show some off-centering effects for BLRGs in the multiple-LRG systems, irrespective of the magnitude difference 
between the BLRG and FLRG. 

The single-LRG systems, each of which contain only one LRG (see Table 1), constitute about 95 percent of the LRG-inferred halos. 
Although the LRG in single-LRG systems might have the off-centering effect, we do not have any other proxy of halo center besides the 
LRG position, unlike in the multiple-LRG systems. Nevertheless, we can estimate the amount of possible off-centering effect as follows. 
Fig. 8 shows the measured LRG-galaxy lensing and the measured cross-correlation of LRGs with photometric galaxies for the single-LRG 
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Figure 7. Similar to the previous figure, but comparing the 68 and 95 per cent C.L. marginalized constraint contours obtained from the LRG-galaxy weak 
lensing measurements (dashed contours) and the LRG-photometric galaxy cross-correlation (solid). 



systems. As in Figs. 4 and 5, we compare the halo model with the measured correlations, and show the best-fit halo model predictions 
including contribution of the off-centered LRGs. In this model fitting, we employ several simplified assumptions; we assumed a narrow mass 
bin for the host halos as well as a thin redshift bin of LRGs, and assumed that the dark matter and the photometric galaxies in the single-LRG 
systems both follow an NFW profile. The LRG-galaxy lensing gives Misot — (0.42 ± 0.04) x 10 14 M@ /h for the best-fit host halo mass, 
which is less massive than the host halo mass of Misot — 1.6 x 10 14 Mq / h for the multiple-LRG systems. However, the constraints on 
the off-centering parameters are not as robust as those for the multiple-LRG systems, because we cannot compare the measurements for the 
different halo center proxies and the derived constraints are more degenerate with the NFW profile parameters. Even so, the cross-correlation 
gives non-zero off-centering parameters: q ccn — 0.76 ± 0.06 and R s — (0.26 ± 0.03) Mpc/h. The LRG-galaxy lensing gives consistent 
results, but with larger error bars due to the smaller signal-to-noise ratio. 

In Fig. 9, we estimate the possible residual FoG contamination in the LRG power spectrum. Shown here is the angle-averaged redshift- 
space power spectrum for the BLRG center relative to the underlying true halo power spectrum, where we have assumed the linear Kaiser 
redshift-space distortion effect. Here the shade region shows the allowed range of the residual FoG effect according to the measurements 
shown in this paper. The upper solid curve is the FoG effect if only BLRGs in the multiple-LRG systems (and not any of the single-LRG 
systems) have the off-centering effects, i.e. the FoG effect, as we found in Fig. 2 15 . Hence, this is a lower limit on the residual FoG effect 
that should exist in the BLRG power spectrum. The lower dotted curve is the worst-case scenario, which shows the FoG contamination if 

15 One might notice that the amplitude of the FoG suppression in Fig. 9 differs from that in Fig. 2. This is because we show the fractional difference of the 
BLRG power spectrum relative to the spectra for the Mean center or the halo center in Figs. 2 and 9, respectively. Since the power spectrum for the Mean-center 
also has the FoG suppression as we have shown, the FoG effect for the BLRG spectrum appears to be larger, when plotted with respect to the spectrum for the 
halo center in Fig. 9 than to the spectrum for the Mean center in Fig. 2. 
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Figure 8. The measured LRG-galaxy weak lensing (Left panel) and the cross-correlation of LRGs with photometric galaxies (Middle) for the single LRG 
systems. In each panel, the different curves show the best-fit halo model predictions including the contributions from the centered and off-centered LRGs, as 
in Figs. 4 and 5. However, in the model fitting we employ several simplified assumptions for the fitting for simplicity: we assume a narrow mass bin for the 
host halo and a thin redshift bin for the single LRG systems, and assume that the dark matter and the photometric galaxies in the host halos follow an NFW 
profile. The LRG-galaxy lensing gives Migof> — (0.42 ± 0.04) X 10 14 Mq/Ii for the best-fit average mass for the host halos, which is smaller than that for 
the multiple-LRG systems, Mig U i, ~ 1.6 X 10 14 Mq//i (see Table 2). Right panel: the best-fit off-centering parameters obtained from the cross-correlation 
measurement are ij cen = 0.76 ± 0.06 and R g = (0.26 ± 0.03) Mpc/h for the fraction of central LRGs and the typical off-center radius, respectively. A 
non-zero off-centering effect is detected at > 2<r significance, and the amount is smaller than in the multiple-LRG systems. Note that the best-fit parameters 
for the LRG-galaxy lensing are consistent with the cross-correlation results within the error bars, but have larger error bars. 



BLRGs in the single-LRG systems also have the off-centering effects in the host halos. To compute this curve, we used the method developed 
in Hikage et al. (2012). To be more precise, we assumed that the off-centering radius R a(M) scales with the virial radius of host halos as 
Roh(M) oc ri8ob(Af), and determined the proportional coefficient so that it reproduces the measurement results for the host halo mass of 
multiple-LRG systems, Migob — 1.6 x 1O 14 M0 /h. In addition, we assumed that the fraction of central BLRGs scales with the halos mass as 
<lc^ RG = 0.6 + 0.05 ln(M/Mi 80 6,wL), where the normalization is determined so as to give q^ a = 0.6 at M 180ti wL = 1-6 x 10 14 M Q /h, 
and the slope 0.05 is taken from the mock galaxy catalog used in Johnston et al. (2007). The genuine FoG effect in the BLRG power spectrum 
would be inside the shaded region. The thick-dashed curve shows the FoG effect implied from the correlation measurements of the single- 
LRG systems in Fig. 8. The dark shaded region around that curve denotes the range covered by varying the off-centering parameters within 
the lcr errors. This result might be closer to the genuine effect. However, since the off-centering constraints for the single-LRG systems are 
derived by employing several additional assumptions as we mentioned above, we believe that the entire range within the shaded grey region 
is still allowed. 

Thus, Fig. 9 implies that the possible FoG effect can be a systematic error in estimating cosmological parameters if the residual FoG 
effect is ignored in the model fitting. For example, we show the effect of massive neutrinos on the halo power spectrum as the dashed curve 
(assuming that the sum of neutrino masses is 0.11 eV, close to the lower bound of the inverted mass hierarchy), but fixing other cosmological 
parameters to the fiducial values. Since the massive neutrinos cause a suppression in the measured power spectrum, as does the FoG effect, 
the figure implies that neglecting the FoG effect might cause a serious systematic error in the derived neutrino mass, with the severity of the 
issue depending on what scales are used. 



5 DISCUSSION AND CONCLUSIONS 

In this paper, we combined three observables (the redshift-space LRG power spectrum, LRG-galaxy weak lensing, and the projected cross- 
correlation of LRGs and photometric galaxies) to constrain the fraction of satellite LRGs and the average off-centered distribution of the 
satellite LRGs in the host halos. When doing so, we focused on multiple-LRG systems, about 4000 systems in total (4.5 per cent of all 
LRG-inferred halos, see Table 1), and compared the measurements obtained by using the different halo center proxies: the positions of the 
brightest LRG (BLRG), the faintest LRG (FLRG), and the mean positions (Mean) in each multiple-LRG system (Figs. 2 and 3). All three 
observables consistently imply that some fraction of LRGs are satellites - even those that are the brightest in the halo - and have offsets from 
the true center. Among the three measurements, the projected cross-correlations of the LRG-inferred halo centers with photometric galaxies 
show the strongest evidence for off-centered LRGs. Perhaps most intriguing or surprising is that some of the BLRGs are satellites and have 
offsets from the halo center, and the Mean positions tend to be closer to the true center, showing the smallest off-centering effect. 

By comparing the halo model developed in Section 2 with the measurements, we constrained the model parameters including the fraction 
of satellite LRGs and the average off-centering radius (Table 2, and Figs. 2, 4 and 5). A brief summary of our findings is as follows. About 60 
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Figure 9. Expected FoG suppression of the angle-averaged, redshift-space power spectrum of LRGs using the BLRG center proxy. Here, we show a possible 
range of the residual FoG contamination relative to the underlying halo power spectrum. The upper solid curve of the shade region shows the FoG effect 
assuming that only BLRGs in the multiple-LRG systems have the off-centering effects, as in Fig. 2. The lower dotted curve shows the FoG effect assuming 
that BLRGs in the single LRG systems also have off-centering effects, computed based on the halo model in Hikage et al. (2012). For this case, we assumed 
that the off-centering radius scales with the virial radius of host halos as R a ff oc ri8n(>(Af), and the FoG amplitude is normalized so as to reproduce the 
FoG effect at the host halo mass of the multiple LRG systems, Migo^ = 1.67 X 10 14 Mq/H, For the fraction of central BLRGs in each halo, we assumed 
that it varies with host halo mass as qf^ 10 = 0.6 + 0.05 ln(Af/M lgoi , WL ), and normalized to q ccn = 0.6 at M 180b = 1.6 X 10 14 M Q //i. Thus 
the dotted curve gives the worst case scenario for the residual FoG effect (see text for details). The bold dashed curve is the FoG effect implied from the 
constraints on off-centering parameters for the single-LRG systems shown in Fig. 8, and the dark shaded region around the curve implies the range covered by 
lcr uncertainties of the off-centering parameters. For comparison, the dashed curve shows the suppression effect caused by massive neutrinos with total mass 
f v = 0.01 (rriy^ot =0.104 eV). This reveals that the possible FoG effect suggested by the results in this paper can be a serious source of systematic errors in 
estimating cosmological parameters using the LRG power spectrum without directly modeling the effect. 



and 30 per cent of the BLRGs and FLRGs, respectively, are central galaxies in the host halos of the multiple-LRG systems, and the remaining 
LRGs are satellite LRGs. If we do not employ a mixture of central and satellite LRGs, we cannot self-consistently model the measurements 
using different choices for the halo centers. The typical off-centering radius is about 400 kpc/ft for both the satellite BLRGs and FLRGs. We 
also confirmed that the Mean centers just happen to be statistically closer to the true center than the BLRG- and FLRG-preferred centers. The 
LRG-galaxy weak lensing for the Mean centers shows a clear dilution effect on small scales. A similar off-centering radius for the satellite 
BLRGs and FLRGs might be reasonable, because the FLRGs and BLRGs are both luminous, early-type galaxies, and therefore should have 
a similar age and history of dynamical friction in the host halos; moreover, the luminosity gap is typically small. The only difference in the 
off-centering effects arises from the different fractions of central LRGs among the BLRG and FLRGs. 

In these constraints, weak lensing plays a very complementary and unique role, although the signal-to-noise ratio is low due to the 
small number density of background galaxies in the shallow SDSS catalog. The mean mass of the host halos of the multiple-LRG systems is 
constrained to be about 1.6 x 10 14 Migob/h (with a roughly 10 per cent statistical error); this is indeed more massive than the mean halo 
mass of all the LRGs. Such large offsets of bright galaxies in massive halos was also suggested recently by Sehgal et al. (2012), based on a 
study of the stacked Sunyaev-Zel'dovich (SZ) effect via a cross-correlation of the SDSS optically-inferred clusters with the CMB map of the 
Atacama Cosmology Telescope survey. They noted that one explanation for the unexpectedly smaller SZ signal is large off-centering effects. 
It would be worth exploring whether our results can resolve the stacked SZ result. 

Based on these constraints, we explored the off-centering effects on the underlying redshift-space power spectrum of LRG-inferred 
halos, and showed the allowed range for the residual FoG contamination in the LRG power spectrum (Fig. 9). As the most optimal case, 
the residual FoG effect is only from the multiple-LRG systems, as we have studied in this paper. As the worst case scenario, by assuming 
that a majority of LRGs in the (far more numerous) single-LRG systems have similar off-centering effects as in the multiple-LRG systems, 
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we estimated the possible maximum, residual FoG effect on the LRG-inferred power spectrum, based on the method in our earlier paper 
(Hikage et al. 2012). The genuine amount of residual FoG effect should be between the two cases we show. In particular, the residual FoG 
effect, if not accounted for, may cause a potentially significant bias in cosmological parameters derived from the measured power spectrum 
amplitude, e.g. the sum of neutrino masses which likewise causes a suppression in the power spectrum amplitude at small scales. Our results 
in Fig. 9 imply that the bias may not be that significant if using the power spectrum amplitude only up to fcmax — 0.1 ft/Mpc, but may be 
quite significant if using the information at larger k. Since the FoG effect is still very difficult to reliably model from first principles, the 
empirical approach we developed in this paper will be useful for correcting the residual FoG effect, or at least making a sanity check of the 
residual FoG effect. 

There are some limitations in the FoG correction for the SDSS LRG catalog. First, the statistical accuracy of the LRG-galaxy lensing 
measurement is limited by the small number density of background galaxies (about 1.2 arcmin~ 2 for the SDSS photometric galaxies), and 
also by the typical SDSS seeing (typically 1.2" FWHM). Second, we lack a good indicator of the halo center for each single LRG system. 
If deeper imaging data is available, it may help us to identify member galaxies around each LRG and then estimate the richness of each 
LRG system, e.g. by using optical richness or the number of member galaxies. In addition, since LRGs are selected by a well-tuned color 
cut, there may be more luminous galaxy(ies) in some of LRG systems, e.g., blue-color brightest cluster galaxy. If such richness information 
on member galaxies or other luminous galaxies are available, we can use the method in this paper for all the LRG systems by comparing 
the correlation measurements for the different halo center proxies. Then, we may be able to correct the residual FoG effect more accurately 
for all the LRG systems. The Subaru HSC survey is planning to have an overlap with the SDSS region, for about 1500 square degrees. The 
number density of galaxies behind z ~ 0.3 LRGs, usable for the shape measurement analysis, is about 20 arcmin~ 2 , a factor of almost 20 
larger than for SDSS. Although the area of the HSC survey is about a factor of 7 smaller than that of the SDSS (about 10000 deg 2 ), the HSC 
can improve the measurement accuracies by more than a factor of -^/20/7 = 1.7 at each radial bin, due to the additional gain in the lensing 
efficiency based on the higher mean redshift of source galaxies than in the SDSS. Since LRGs in the 1500 sq. degrees area can be safely 
considered as a fair, representative sample of LRGs, the HSC survey would improve the accuracy of the FoG correction for the entire SDSS 
LRG sample, and will more generally increase our understanding of the nature of LRGs and their relation to their host halos. Furthermore, 
the higher number density of imaging galaxies can also improve the accuracies of the projected cross-correlation measurements. 

The deeper depth of an imaging survey such as the HSC survey is also very useful to extend the method of this paper to massive 
galaxies at higher redshifts, which is being surveyed by the SDSS-III BOSS survey. BOSS has a denser sampling of massive galaxies by 
a factor of 3-4 compared to the SDSS, so the FoG contamination can be more significant. Again, by having background galaxies around 
each BOSS galaxy and/or photometric galaxies in the BOSS galaxy redshift range (z — [0.4, 0.75]), we can use weak lensing and cross- 
correlation measurements to observationally explore the nature of those massive red galaxies and to remove the FoG contamination for doing 
cosmology. These are all crucial in order to reliably use the amplitude and shape information of the galaxy power spectrum in order to 
constrain the growth rate as well as cosmological parameters such as the neutrino masses. 

Our paper demonstrates the complementarity of spectroscopic and imaging surveys for observationally constraining the connection 
between galaxies and dark matter halos. This is just one example, and there are various synergies available, if the imaging and spectroscopic 
surveys see the same region of the sky (or have at least a sufficiently large overlapping area). This is the case for the SDSS BOSS and the 
HSC survey, the HSC and PFS surveys, and the Euclid survey. First, as we stressed in this paper, we can measure the galaxy-galaxy lensing 
or the cross-correlation as a function of the projected radius R in the comoving unit from the spectroscopic galaxies, rather than the angular 
separation. The correlations do not mix different scales at different redshifts, and can preserve the same physical radius even if taking a 
wider redshift slice in the correlation analysis, e.g., the virial radii of halos or the BAO scales. Hence, such correlation functions measured 
against the projected radii R yield a measurements of the three-dimensional power spectrum or correlation functions (see Eqs. 9 and 14). 
This is obvious, but has not been explicitly stressed in the literature. Furthermore, combining the imaging and spectroscopic surveys allows 
for a reconstruction of galaxy bias functions on large scales as well as a calibration of other systematic errors such as the halo mass, photo- z 
errors, and shear multiplicative biases (Reyes et al. 2010; Baldauf et al. 2010; Oguri & Takada 201 1; Mandelbaum et al. 2012b). Thus various 
synergies between imaging and spectroscopic surveys need to be more carefully explored in order to attain the full potential of these surveys 
for deriving the most stringent and robust constraints possible. 



ACKNOWLEDGMENTS 

We thank Shirley Ho, Issha Kayo, Surhud More, Beth Reid, Uros Seljak and Kazuhiro Yamamoto for useful discussion and valuable com- 
ments. CH acknowledges support from a Japan Society for Promotion of Science (JSPS) fellowship. RM acknowledges the support of the 
Department of Energy Early Career Award program for some of the duration of this work. MT greatly thanks Department of Astrophysical 
Sciences, Princeton University for its warm hospitality during his visit, where this work was done. DNS and CH acknowledge support from 
NSF grant AST-0707731 and the NASA AST theory program. DNS thanks the IPMU for its warm hospitality during his visit, where the work 
was completed. This work is in part supported in part by JSPS Core-to-Core Program "International Research Network for Dark Energy", by 
Grant-in- Aid for Scientific Research from the JSPS Promotion of Science, by Grant-in- Aid for Scientific Research on Priority Areas No. 467 
"Probing the Dark Energy through an Extremely Wide & Deep Survey with Subaru Telescope", by World Premier International Research 



© 0000 RAS, MNRAS 000, 000-000 



20 Hikage, Mandelbaum, Takada & Spergel 




Figure Al. Left panel: the cross-correlation functions of the BLRG or FLRG position with the photometric galaxies, for the two subsamples of the multiple- 
LRG systems, where the multiple-LRG systems are divided into halves. One subsample is defined by the systems which have the larger magnitude difference 
between BLRG and FRLG than the median (larger AM), while the other contains the remaining systems, those with the smaller magnitude difference (smaller 
AM ). For each subsample, we measured the cross-correlations for the BLRG or FLRG centers as indicated by the legend. Note that the correlation amplitudes 
are not reliable due to the dilution effect caused by photo-2 errors. Right panel: the best-fit off-centering parameters obtained from each sub-sample. Since we 
cannot obtain sufficiently tight constraints on the parameters due to insufficient statistical power, we fixed the typical off-centering radius to R = 400 kpc/h 
for both the BLRG and FLRG centers, and constrained the fraction of central BLRGs or FLRGs, gJf C n RG or 9^ C n RG . The fraction of the central BLRGs is not 
different for the two subsamples, but the fraction of the central FLRGs is smaller for the subsample with the larger magnitude difference. 



Center Initiative (WPI Initiative), MEXT, Japan, by the FIRST program "Subaru Measurements of Images and Redshifts (SuMIRe)", CSTP, 
Japan, and by the exchange program between JSPS and DFG. 



APPENDIX A: DEPENDENCE OF THE OFF-CENTERING EFFECTS ON THE MAGNITUDE DIFFERENCE BETWEEN 
BLRG AND FLRG 

We found that some BLRGs in the multiple-LRG systems are satellite galaxies. As shown in Fig. 1, the BLRG in some systems is brighter 
than FLRG by only a few tenths of a magnitude, implying that BLRG and FLRG are not so different. Given this fact, one might consider that 
satellite BLRGs are mainly from such multiple-LRG systems where the FLRG is instead a central galaxy and the BLRG is a satellite. On 
the other hand, for systems where the BLRG and FLRG luminosities are significantly different, the BLRG may have a higher chance to be 
central. In this appendix, we study this possibility by dividing the multiple-LRG systems into halved subsamples in terms of the magnitude 
difference between BLRG and FLRG in each system. One halved subsample contains the systems for which the BLRG and FLRG have a 
smaller magnitude difference than the median (the "small AM" subsample), while the other subsample contains the remaining systems (the 
larger "AM"). 

The left panel of Fig. Al shows the measured cross-correlations for the halved subsamples, using either BLRG or FLRG centers. The 
cross-correlations for the different subsamples and centers differ from each other, especially in their amplitudes. However, as we stressed in 
Section 2.4, we cannot directly compare the correlation amplitudes because the amplitude suffers from a dilution due to photo-z errors. Instead 
we need to use the shape of the correlation function to infer the off-centering effects. By fitting the halo model to the cross-correlations, we 
estimate the off-centering parameters. However, due to the insufficient statistical accuracies of the halved subsample, we were not able to 
obtain tight constraints on the parameters, after marginalizing over other parameters. Hence, in the right panel, we show the constraints on 
the fractions of central BLRGs or FLRGs for each of the two subsamples when fixing the off-centering radius to 0.4 Mpc//i, the best-fit 
value for all the multiple-LRG systems. The plot shows that the fraction of central BLRGs is similar for the two subsamples, qf^ G — 0.5, 
but the fraction of central FLRGs is higher for the subsample with the smaller magnitude difference between BLRG and FLRG than the 
other subsample, supporting the hypothesis that the FLRG is not so different from the BLRG, and the FLRG is instead a central galaxy in the 
subsample. 
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